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Abstract. This paper presents a rule-based task-oriented dialogue system for second 
language learning and a knowledge extraction method which automatically extracts 
the training data for Natural Language Understanding (NLU) and dialogue rules for 
dialogue management from a Dialogue Map (DM). The DM consists of turn-by- 
turn utterances between the system and the learner. Therefore, the proposed method 
can automatically extend a new dialogue domain by constructing a dialogue map 
with a simple format. We constructed two dialogue maps for English and Korean, 
respectively, and implemented English and Korean task-oriented dialogue systems 
using the DMs. In the experiments, although the turn success rates are relatively 
low (78.1% in English and 78.76% in Korean), the task success rates are 90.83% in 
English and 99.17% in Korean. The systems constructed by the proposed method 
should enable learners to communicate successfully in the topic despite some 
mistakes in the system responses. 


Keywords: knowledge extraction method, task-oriented dialogue system, second 
language learning, dialogue map. 


1. Introduction 


Speech dialogue system technology is a promising tool to use in Computer-Assisted 
second Language Learning (CALL). Through dialogue with the system, we expect 
to improve learners’ second language skills. Generally, the dialogue system suitable 
for CALL can be viewed as a Task-oriented Dialogue System (TDS) rather than 
a chatbot system, as it is necessary to be able to converse with the subject in a 
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specific situation (Johnson & Valente, 2009). Most TDSs are implemented as a 
slot-filling dialogue model which is not suitable for language learning because a 
slot-filling dialogue consists of only simple request and response pairs. 


We already proposed a Task Graph Dialogue Model (TGDM) that enables topic 
conversation in various situations (Choi, Kwon, Kim, & Lee, 2016). The model 
can divide a complex topic task into smaller subtasks and define the order between 
them. In TGDM, subtasks are defined as nodes and the order as edges. We also 
suggested a DM based on TGDM, which can be constructed by educators who do 
not know the dialogue systems (Choi et al., 2016). 


In this paper, we introduce a knowledge extraction method platform that 
automatically convert a DM into the knowledge of a TGDM-based TDS to create 
a dialogue system for a new topic. 


2. Automatic knowledge extraction 
for TGDM-based TDS 


2.1. TGDM 


TDS needs a dialogue model that enables dialogue between learners and the 
system to perform a specific task. The dialogue model defines the dialogue 
strategy and response of the system to a learner’s utterance on the dialogue 
context. In the model, the utterances and the dialogue context are represented by 
dialogue acts and slots. A dialogue act is a function of utterance and a slot is a 
semantic concept in the conversation. For example, in ‘ordering food task’, the 
utterance J want to have a coffee is represented by want_a_food(beverage= ‘coff 
ee’), which consists of the dialogue act want_a_food, as well as the beverage slot 
and its value coffee. 


The proposed TGDM divides a complex task into small subtasks and places order 
constraints amongst the subtasks. TGDM is similar to finite-state transition models. 
Dialogue models of individual subtasks are implemented as a hybrid dialogue 
model of a slot-filling and information state. Therefore, TGDM is hybridised with 
different models, where expression ability of TGDM is more powerful than other 
models. Also, TGDM can (1) have the advantage that subtasks can be used for 
other tasks and (2) analyse learner’s utterance and generate its system response 
according to subtasks. 
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2.2. DM for T@DM 


DM is a method of expressing TGDM in a dialogue script manner. Figure | shows 
the ordering food task in the snack bar as DM. The ordering food task is divided into 
six subtasks that consist of turn-by-turn dialogue. The diversity of the conversation 
flow is made by the learner’s various responses to a system utterance. In DM, 
each terminal turn of the subtask contains the next subtask information (for further 
details of DM, see Choi et al., 2016). 


Figure 1. An example of DM for ‘ordering food task’ 


Task: ordering food 


Subtasks: Greet > Choosing L Choosing Choosing L,| Asking L»! Saying 


main dishes side dishes beverages bill good bye 


(S1) System: What would you like to order? 
- What would you like to have for your main dish? 
- What would you like? 


(U1) User: I would like to have the @maindishes. @maindishes={‘steak and eggs”, ‘toasts} 
- I'll have the @maindishes. 


-_|want @maindis 


(U2) User: : | would like to have the @maindishes. @maindishes={‘omelette} 
(S2) System: We have two kind of @maindishes, mushroom and cheese. Which one would like you? 


'”[[(U3) User: : | would like tohave the @omelette type. @omelette_type={mushroom’, “cheese’} || Goto: 
Choosing 
side dishes 


(Un) User: : What do you have? 


[| (St) System: We have five kinds of main dishes, omelette, pancakes, sandwich, toasts, waffle, and steak 
and eggs. Which one would you like? 


I (U1) User: | would like to have the @maindishes. @maindishes={‘steak and eggs’, ‘toasts } 


2.3. Automatic knowledge extraction using DMs 


In this section, we introduce a method that can build a dialogue system for a 
new task by writing a machine readable DM. Figure 2 shows an overview of our 
TDS and also the procedures of automatic knowledge extraction method which 
automatically extracts the knowledge needed by our TDS from the DM. 


Our TDS consists of a Structured Support Vector Machine (SSVM) based NLU 
(Kwon, Lee, Kim, & Lee, 2015), task graph-based dialogue management, and a 
generation template based Natural Language Generation (NLG). First, SSVM- 
based NLU analyses the learner utterance and determines the intention using 
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SSVM training data. Then, task graph-based dialogue management locates the 
appropriate system response for the learner intention on the context using dialogue 
tules and updates the context. Finally, NLG generates the system utterance from a 
template of the selected system response. 


Figure 2. Automatic knowledge extraction for TGDM-based TDS using DM 


Automatic knowledge extraction using dialogue map 


educator 


@ Dialogue Map 
[oe] 


Generation Template 
have_two_kind(@maindish 


: ¥ - es=value) > 
Dialogue intention generation We have two kinds of 


@maindishes, mushroom 
and cheese. Which one 
would you like? 


Taining 
corpus for user 
dialogue 
intention 


¥ 


Rule extraction for task and 
SSVM based system response 
NLU training 


Sentence Generation 


SSVM training DB Template DB 


Dialogue Rule DB 


user/learner 
| want to have omelette. SSVM based Task graph Template based 
Natural Language based Dialogue Natural Language 


| 


We have two kinds of 
omelette, mushroom and 
cheese 

Which one would you like? 


Understanding Management Generation 


would _like_to_have(ma have_two_kind(@ma 
indishes=omelette*) indishes="omlette’) 


Task graph based task-oriented dialogue system 


Automatic knowledge extraction methods using DMs consist of dialogue 
intention generation, SSVM based NLU training, and rule extraction. Dialogue 
intention generation automatically makes the intention for each utterance in 
DMs using its predicate and object, which are analysed by shallow parser. If 
the utterance has no object or the object is a slot, it sets the dialogue act with 
the predicate, otherwise, with a combination of the predicate and object. For 
example, Utterances U1, U2, and U3 of Figure | have the same predicate would 
like to have, but have different objects. So, the intention generation makes the 
intention would_like_to_have(@maindishes=‘value’) for U1 and U2, and the 
intention would_like_to_have(@omelette_type=‘value’) for U3. 


SSVM training data consists of triples of user sentences of DMs, slot-annotated 
sentences, and its intentions automatically generated by dialogue intention 
generation. The SSVM-based NLU training module trains slot classification using 
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sentences and their slot-annotated sentences, as well as intention classification 
using sentences and their intentions. 


Dialogue rules are generated after dialogue intention generation. The rules are 
divided into subtask movement rules and turn-by-turn rules. Subtask movement 
rules consist of a movement condition and a subtask information for the current 
subtask and the next subtask. The movement condition is defined as the terminal 
utterance’s intention and/or all slots and their values as appear in the path from the 
initial utterance to the terminal utterance. 


Turm-by-turn rules consist of activating the condition and the action to be 
performed when the condition is satisfied. The rules are extracted from each turn- 
by-turn utterance. Its condition is constructed by the subtask information, previous 
intention, and its talker (system or user/learner), and its action is current intention 
and its talker. From the rules, we may determine a proper system response intention 
to the user/learner’s input on the context, and also proper utterances of the next 
user/learner’s turn to current system response. 


Sentence generation templates are easily generated from the utterance pattern of 
the DM. The templates are classified by the intention and the subtask. 


3. Preliminary experiments and discussion 


We implemented English and Korean TDSs. The language educators constructed 
two DMs and implemented two TDSs using the DMs through the proposed 
knowledge extraction method for each language. To evaluate the performance, we 
recruited 20 second language learners for each language. The learners freely talked 
on the given task and conducted four conversations with each other topic on each 
task. 


Table | shows the experiment results. The turn success rates to evaluate whether 
the system responded correctly to learners’ utterances are 78.1% in English and 
78.76% in Korean, and the task success rates to evaluate whether learners completed 
the conversation with the system in the topics are 90.83% in English and 99.17% 
in Korean. There were little differences in the performance depending on the 
number of learners’ conversation attempts. According to these preliminary results, 
we believe the systems constructed by writing DMs may have the performance to 
communicate successfully with learners in the topic despite a rather low success 
rate; this will be further explored. 
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Table 1. The experiment results 


Language | Task Task success rate % Turn success rate % 
English ordering food 88.33 78.74 

city tour ticketing 93.33 7745 

both 90.83 78.10 
Korean ordering Korean food 100 81.41 

clothes shopping 98.33 75.65 

both 99.17 78.76 


Task success rate (Y%) = (number of task completed dialogue) / (total number of dialogue) x 100 
Turn success rate (%) = (number of correct system’s responses to learners’ utterances) / (total number of learners’ utterances) 
x 100 


4. Conclusions 


This paper describes an automatic knowledge extraction method developing TDS 
using the DM. Through the experiments, the performance of the TDS constructed 
by writing a DM showed that second language learners could successfully practice 
conversations on a given topic. With the proposed automatic knowledge extraction 
method, we expect that anyone could construct dialogue systems for language 
learning. In future research, we will investigate whether educators can build DMs 
quickly and easily and collect more data to further improve the system’s success rate. 
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